- 4046 Views
- 4 replies
- 3 kudos
ASP1.2 Error create database in Spark Programming with Databricks training
I'm on Demo and Lab in Dataframes section. I've imported the dbc into my company cluster and has run "%run ./Includes/Classroom-Setup" successfully. When i run the 1st sql command %sql CREATE TABLE IF NOT EXISTS events USING parquet OPTIONS (path "/m...
- 4046 Views
- 4 replies
- 3 kudos
- 3 kudos
I had the same issue and solved it like this:In the includes folder, there is a reset notebook, run the first command, this unmounts all mounted databases.Go back to the ASP 1.2 notebook and run the %run ./Includes/Classroom-Setup codeblock.Then run ...
- 3 kudos
- 3276 Views
- 1 replies
- 0 kudos
SFTP - JSchException: Algorithm negotiation fail
When I tried to read the SFTP (CSV file) in Databricks I'm getting the below error"JSchException: Algorithm negotiation fail"Code:var df = spark.read.options(Map("header"->"true","host"->"20.118.190.30","username"->"user","password"->"pass","fileForm...
- 3276 Views
- 1 replies
- 0 kudos
- 0 kudos
@MonishKumar Could you provide the entire exception ?From the one line error message, I suspect this is due to the SSL cipher suites required by the SFTP server is not available on cluster. You can run below to get the cipher suites that sftp require...
- 0 kudos
- 2216 Views
- 1 replies
- 0 kudos
Delta Live Table Graph different with no change in Notebook code
I have a DLT code to create 40+ bronze tables. The tables are created on top of the latest parquet files for each of those tables. While executing the pipeline, sometimes I notice that the graph is different than the regular one i see. I do not under...
- 2216 Views
- 1 replies
- 0 kudos
- 0 kudos
@Retired_mod - Thank you for your response. There is no change in the table dependencies. The code to create the individual raw tables look like this: The input to this is always the same 40 tables with only the underlying parquet file changing. I c...
- 0 kudos
- 1686 Views
- 0 replies
- 0 kudos
merge breaking persistance of dataframe
Hi allIn the minimal example below you can see that executing a merge statement trigger recomputation of a persisted dataframe. How does this happen? from delta.tables import DeltaTable table_name = "hive_metastore.default.test_table" # initializ...
- 1686 Views
- 0 replies
- 0 kudos
- 3426 Views
- 2 replies
- 1 kudos
I would like to know why I am getting this error when I tried to earn badges for lakehouse fundamen
I would like to know why I am getting this error when I tried to earn badges for lakehouse fundamentals. I can't access the quiz page. Can you please help on this?Getting below error:-403FORBIDDENYou don't have permission to access this page2023-08-...
- 3426 Views
- 2 replies
- 1 kudos
- 8888 Views
- 2 replies
- 0 kudos
Resolved! how to set jobs permission with rest api
create job with cli, but can not set the permission with cli,have to use rest api to set permission:https://docs.databricks.com/api/workspace/permissions/setbelow is my command in windows to set permission:curl -X PUT https://my-workspace-url.azureda...
- 8888 Views
- 2 replies
- 0 kudos
- 0 kudos
thank you, the new permission list should be a whole list, not the new permission
- 0 kudos
- 1146 Views
- 0 replies
- 0 kudos
the function base64 has issue with function aes_encrypt and secret
for the below query:select base64( aes_encrypt( '00', to_binary(secret('secret-scope', 'data-protection-secret'), 'BASE64') ) )when the value to encrypt is 2 characters, the return is always "REDACTED_POSSIBLE_SECRET_ACC...
- 1146 Views
- 0 replies
- 0 kudos
- 1511 Views
- 0 replies
- 0 kudos
Looking for Upload file to dbfs using "/api/2.0/dbfs/put"
Hello,I am trying to upload the file from local linux machine to dbfs using request.post(<URI>,<Headers>, params={"path": "dbfs:/tmp", "contents": local_path}) and getting the error b'{"error_code":"INVALID_PARAMETER_VALUE","message":"You must provid...
- 1511 Views
- 0 replies
- 0 kudos
- 3146 Views
- 2 replies
- 1 kudos
Backup/Export Databricks SQL Column Comments
We've had users make comments on tables/columns throughout Databricks SQL using the Data Explorer UI. I'm looking for a way to backup these comments, but when I run DESCRIBE TABLE, the comment column is always null despite being non-null in Data Expl...
- 3146 Views
- 2 replies
- 1 kudos
- 1 kudos
@804082 - Markdown does not render when returned by DESCRIBE statements. we can view them on the Data Explorer UI.Reference: https://docs.databricks.com/en/data/markdown-data-comments.html#document-data-with-markdown-comments
- 1 kudos
- 1526 Views
- 1 replies
- 0 kudos
Databricks Job RUns API
/api/2.1/jobs/runs/list Currently returns all jobs which got executed within specified time lines which we provide as input. Is There any way where we can get only specific jobs as per their status(only success)?
- 1526 Views
- 1 replies
- 0 kudos
- 0 kudos
@venkat94 thanks for the feedback. We are working on updating the Jobs Runs API so you can filter runs by status e.g. only success. Stay tuned in the next couple of months.
- 0 kudos
- 2151 Views
- 1 replies
- 0 kudos
Re-running DLT Pipeline Does Not Add Data After Delete
I am using DLT and unity catalog and using managed tables. The first table in this pipeline is a live streaming table. I first did this in the SQL editor:DELETE FROM my_table;This appears to have deleted all the records, which I wanted since now when...
- 2151 Views
- 1 replies
- 0 kudos
- 0 kudos
@Mo is correct!
- 0 kudos
- 2275 Views
- 0 replies
- 0 kudos
SQL Merge condition
Hi guys I have a question regarding this merge step and I am a new beginner for Databricks, trying to do some study in data warehousing, but couldn't figure it out by myself. need your help with it. Appreciate your help in advance. I got this questi...
- 2275 Views
- 0 replies
- 0 kudos
- 2675 Views
- 1 replies
- 0 kudos
Azure Shared Clusters - P4J Security Exception on non-whitelisted classes
When I try to use a third party JAR on an Azure shared cluster - which is installed via Maven and I can successfully import - , I get the following message: py4j.security.Py4JSecurityException: Method public static org.apache.spark.sql.Column com.da...
- 2675 Views
- 1 replies
- 0 kudos
- 0 kudos
Thanks Kaniz.I must use a shared cluster because I'm reading from a DLT table stored in a Unity Catalog.https://docs.databricks.com/en/data-governance/unity-catalog/compute.htmlMy understanding is that shared clusters are enforcing the Py4J policy I ...
- 0 kudos
- 3498 Views
- 3 replies
- 1 kudos
Delta live table UC Kinesis: options overwriteschema, ignorechanges not supported for data sourc
I try to build a DLT in UC with Kinesis as producer.My first table looks like: @dlt.create_table( table_properties={ "pipelines.autoOptimize.managed": "true" }, spark_conf={"spark.databricks.delta.schema.autoMerge.enabled": "true"},)def feed_chu...
- 3498 Views
- 3 replies
- 1 kudos
- 1 kudos
If you use the "Preview" Channel in the "Advanced" section of the DLT Pipeline, this error should resolve itself. This fix is planned to make it into the "Current" channel by Aug 31, 2023
- 1 kudos
- 3014 Views
- 0 replies
- 0 kudos
Delta Live Tables maintenance schedule
I have a DLT that runs every day and an automatically executed maintenance job that runs within 24 hours every day. The maintenance operations are costly, is it possible to change the schedule to once a week or so?
- 3014 Views
- 0 replies
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 791 | |
| 553 | |
| 349 | |
| 287 |
