Hi Team,I am building a DLT pipeline and planning to use APPLY_CHANGES from Bronze to Silver. In the bronze table, a column has a json value. This value contains questions and answers as key, value pair and can change depending on list of questions h...
What is the problem?I am getting this error every time I run a python notebook on my Repo in Databricks.BackgroundThe notebook where I am getting the error is a notebook that creates a dataframe and the last step is to write the dataframe to a Delta ...
Hi @Sara Corral​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers y...
I have a spark submit job which is running one python file called main.py.The other file is alert.py which is being imported in main.py.Also main.py is using multiple config files.Alert.py is passed in --py-files and other config files are passed as ...
User sessions automatically timeout after six hours of idle time. This is not configurable like @Kunal Gaurav​ mentioned. Please raise a feature request if you have a requirement to configure this.Now, in Azure you could configure AAD refresh token ...
I'm on Demo and Lab in Dataframes section. I've imported the dbc into my company cluster and has run "%run ./Includes/Classroom-Setup" successfully. When i run the 1st sql command %sql CREATE TABLE IF NOT EXISTS events USING parquet OPTIONS (path "/m...
I had the same issue and solved it like this:In the includes folder, there is a reset notebook, run the first command, this unmounts all mounted databases.Go back to the ASP 1.2 notebook and run the %run ./Includes/Classroom-Setup codeblock.Then run ...
When I tried to read the SFTP (CSV file) in Databricks I'm getting the below error"JSchException: Algorithm negotiation fail"Code:var df = spark.read.options(Map("header"->"true","host"->"20.118.190.30","username"->"user","password"->"pass","fileForm...
@MonishKumar Could you provide the entire exception ?From the one line error message, I suspect this is due to the SSL cipher suites required by the SFTP server is not available on cluster. You can run below to get the cipher suites that sftp require...
I have a DLT code to create 40+ bronze tables. The tables are created on top of the latest parquet files for each of those tables. While executing the pipeline, sometimes I notice that the graph is different than the regular one i see. I do not under...
@Retired_mod - Thank you for your response. There is no change in the table dependencies. The code to create the individual raw tables look like this: The input to this is always the same 40 tables with only the underlying parquet file changing. I c...
Hi allIn the minimal example below you can see that executing a merge statement trigger recomputation of a persisted dataframe. How does this happen? from delta.tables import DeltaTable table_name = "hive_metastore.default.test_table" # initializ...
I would like to know why I am getting this error when I tried to earn badges for lakehouse fundamentals. I can't access the quiz page. Can you please help on this?Getting below error:-403FORBIDDENYou don't have permission to access this page2023-08-...
create job with cli, but can not set the permission with cli,have to use rest api to set permission:https://docs.databricks.com/api/workspace/permissions/setbelow is my command in windows to set permission:curl -X PUT https://my-workspace-url.azureda...
thank you, the new permission list should be a whole list, not the new permission
for the below query:select base64( aes_encrypt( '00', to_binary(secret('secret-scope', 'data-protection-secret'), 'BASE64') ) )when the value to encrypt is 2 characters, the return is always "REDACTED_POSSIBLE_SECRET_ACC...
Hello,I am trying to upload the file from local linux machine to dbfs using request.post(<URI>,<Headers>, params={"path": "dbfs:/tmp", "contents": local_path}) and getting the error b'{"error_code":"INVALID_PARAMETER_VALUE","message":"You must provid...
We've had users make comments on tables/columns throughout Databricks SQL using the Data Explorer UI. I'm looking for a way to backup these comments, but when I run DESCRIBE TABLE, the comment column is always null despite being non-null in Data Expl...
@804082 - Markdown does not render when returned by DESCRIBE statements. we can view them on the Data Explorer UI.Reference: https://docs.databricks.com/en/data/markdown-data-comments.html#document-data-with-markdown-comments
/api/2.1/jobs/runs/list Currently returns all jobs which got executed within specified time lines which we provide as input. Is There any way where we can get only specific jobs as per their status(only success)?
@venkat94 thanks for the feedback. We are working on updating the Jobs Runs API so you can filter runs by status e.g. only success. Stay tuned in the next couple of months.
I am using DLT and unity catalog and using managed tables. The first table in this pipeline is a live streaming table. I first did this in the SQL editor:DELETE FROM my_table;This appears to have deleted all the records, which I wanted since now when...
@Mo is correct!
| User | Count |
|---|---|
| 1629 | |
| 790 | |
| 511 | |
| 349 | |
| 287 |
