- 607 Views
- 2 replies
- 2 kudos
Resolved! "delta-lake" demo fails with"config" notebook not found when running "00-setup"
Hi all,The delta-lake demo used to run fine for us around October 2024. Reinstalling it now, it fails on initialisation.Using runtime version 15.4 on a trial Databricks installation, and executing dbdemos.install('delta-lake', overwrite=True, use_cur...
- 607 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @RoelofvS , I was able to replicate the issue and reached out to the team that maintains dbdemos, they will get this addressed. Until that is addressed, you can try manually creating that `config` notebook as follows: #Note: we do not recommend to...
- 2 kudos
- 645 Views
- 2 replies
- 1 kudos
Resolved! Explode function to flatten the JSON
I've the DDL as below. Create or replace table test ( prices ARRAY<STRUCT<Ord:STRING:,Vndr:STRING,Prc:STRING>> ) using delta location "path" Now I want to flatten the JSON and I've tried as below but it's throwing an error, "[UNRESOLVED.COLUMN.WITH_...
- 645 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @David_Billa ,You can use following from_json function in spark which can convert struct into individual column. Refer this link https://spark.apache.org/docs/3.4.0/api/python/reference/pyspark.sql/api/pyspark.sql.functions.from_json.html.Also, yo...
- 1 kudos
- 249 Views
- 1 replies
- 0 kudos
Managed table storage not accessible in cloud storage
Hi All,I have create a new catalog in the unity catalog which has a cloud location for managed tables. but when accessing the location in Azure portal it denies me to see the files.I want to see all my data files ( parquet / JSON) in the storage acco...
- 249 Views
- 1 replies
- 0 kudos
- 0 kudos
Do you have full permissions on this storage location? Also please remember the best practices for Managed tables: You should not use tools outside of Databricks to manipulate files in managed tables directly. You should only interact with data files...
- 0 kudos
- 1985 Views
- 4 replies
- 1 kudos
drop or alter primary key constraint for a streaming table in delta live tables
I have a dlt streaming table with a primary key constraint defined through the schema definitions. I have redeployed the same dlt pipeline with the same target. When attempting to run the pipeline, I get the error, ErrorClass=RESOURCE_ALREADY_EXISTS...
- 1985 Views
- 4 replies
- 1 kudos
- 1 kudos
The `RESOURCE_ALREADY_EXISTS` error you're encountering suggests that the primary key constraint `pk_constraint_name` already exists in the target delta table. This constraint may have been created during a previous deployment of the DLT pipeline or ...
- 1 kudos
- 1433 Views
- 2 replies
- 0 kudos
Creating Linked Service in ADF to link Databricks Community Edition
Using Current Community Edition I was Unable to create a linked service unable to generate/copy personal Access Token .Tried Installing ODBC but unable to set it properly. Any suggestions / Help would be highly Appreciated
- 1433 Views
- 2 replies
- 0 kudos
- 0 kudos
In community Edition workspaces the PAT token is not allowed, instead of this you might need to generate an Oauth token to be able to authenticate: https://docs.databricks.com/en/dev-tools/auth/oauth-u2m.htmlYou can refer to the ODBC guide in page 19...
- 0 kudos
- 855 Views
- 3 replies
- 0 kudos
Resolved! Deploying Data Source API code
This might be a stupid question but there's just no mention of what to do here. I'm looking at the blog (https://www.databricks.com/blog/simplify-data-ingestion-new-python-data-source-api) and documentation (https://learn.microsoft.com/en-us/azure/d...
- 855 Views
- 3 replies
- 0 kudos
- 1021 Views
- 2 replies
- 0 kudos
Resolved! ODBC connection issue Simba 64 bit driver
Hello AllAm getting the below error when trying to create ODBC DSN Simba 64 in local system to connect Databricks Server using the token and enabled SSL System trust store & Thrift Transport: HTTP. Thanks,[Simba][ThriftExtension] (14) Unexpected resp...
- 1021 Views
- 2 replies
- 0 kudos
- 2106 Views
- 3 replies
- 4 kudos
Databricks SDK for Python: Errors with parameters for Statement Execution
Hi team,Im using Databricks SDK for python to run SQL queries. I created a variable as below:param = [{'name' : 'a', 'value' :x'}, {'name' : 'b', 'value' : 'y'}]and passed it the statement as below_ = w.statement_execution.execute_statement( warehous...
- 2106 Views
- 3 replies
- 4 kudos
- 4 kudos
The following works: response = w.statement_execution.execute_statement( statement='ALTER TABLE users ALTER COLUMN :col_name SET NOT NULL', warehouse_id='<warehouseID>', parameters=[sql.StatementParameterListItem(name='col_name' value='u...
- 4 kudos
- 3567 Views
- 7 replies
- 0 kudos
Cannot login Databricks Community Edition with new account
It seems it is not allowing me to log into databricks community edition. I have recently created a new account and had the account verified. However, whenever I try to log in, I am redirected to the same page without throwing any errors. When I do en...
- 3567 Views
- 7 replies
- 0 kudos
- 0 kudos
Please check steps provided in https://community.databricks.com/t5/support-faqs/databricks-community-sso-august-3rd-2024/ta-p/78459If issue persist please reach out to databricks-community@databricks.com
- 0 kudos
- 392 Views
- 2 replies
- 0 kudos
dbutils.notebook API: pass data back to caller notebook
Hi all, according to this doc, we can pass data back through temp views, DBFS, or JSON data.However, in my case, i need to pass both a temp view, as well as some metadata in JSON. is there a way to exit with BOTH a view AND json, something likedbuti...
- 392 Views
- 2 replies
- 0 kudos
- 0 kudos
I can have a try and see if it's the same like exit(view_name) in that the view is created in global_temp_db and that the lifecycle is with the job compute.
- 0 kudos
- 1031 Views
- 8 replies
- 0 kudos
Not able create table form external spark
py4j.protocol.Py4JJavaError: An error occurred while calling o123.sql.: io.unitycatalog.client.ApiException: generateTemporaryPathCredentials call failed with: 401 - {"error_code":"UNAUTHENTICATED","message":"Request to generate access credential for...
- 1031 Views
- 8 replies
- 0 kudos
- 0 kudos
You need the generateTemporaryPathCredentials API as you are trying to create external tables
- 0 kudos
- 10582 Views
- 3 replies
- 0 kudos
Resolved! token share
Hello community,I want to have a new token, to be available to our users to interact with staging workspace. Is it possible to generate a token to be used for triggering only workflow in staging workspace databricks with databricks api?
- 10582 Views
- 3 replies
- 0 kudos
- 0 kudos
you will need to have your own personal token to create one for your SP, in the UI you need to go to Settings > Under User > Developer > Personal Access Token.Once you have your own token you can run the API I mentioned on my previous post and you ne...
- 0 kudos
- 426 Views
- 5 replies
- 1 kudos
For each task field
Hi community,I was wondering after passing a list of dict through tasks using .taskValue.set() method, how to maintain the same data type through each task?Because seems, that when I use the for loop and getting by the parameters each element of the ...
- 426 Views
- 5 replies
- 1 kudos
- 1 kudos
Yeah, to ensure that the data types are maintained, you can convert the values to the desired types after deserialization. This is necessary because JSON does not distinguish between integers and floats, and all numbers are deserialized as floatsThe ...
- 1 kudos
- 1599 Views
- 3 replies
- 0 kudos
Databricks Upload local files (Create/Modify table)
Hello Team,I believe Databricks come out recently feature of Create or modify a table using file upload which is less than 2 GB (file format CSV, TSV, or JSON, Avro, Parquet, or text files to create or overwrite a managed Delta Lake table) on Self Se...
- 1599 Views
- 3 replies
- 0 kudos
- 0 kudos
For Sharing a CSV file containing PII data with another user who should not have access to PII data elements: You can use Databricks' Unity Catalog to manage and govern access to data. Unity Catalog allows you to define fine-grained access controls a...
- 0 kudos
- 1349 Views
- 4 replies
- 4 kudos
Merge operation to delta table with new column starting with upper case seems to be not working
Hello,I have a simple spark dataframe saved to a delta table:data = [ (1, "John", "Doe"), (2, "Jane", "Smith"), (3, "Mike", "Johnson"), (4, "Emily", "Davis")]columns = ["Id", "First_name", "Last_name"]df = spark.createDataFrame(data, sche...
- 1349 Views
- 4 replies
- 4 kudos
- 4 kudos
I assume you must be facing an error referred here on GitHub issues page. you can follow it, they make release fix for same.[BUG][Spark] issue when merge using autoMerge property · Issue #3336 · delta-io/delta · GitHub
- 4 kudos
| User | Count |
|---|---|
| 1612 | |
| 768 | |
| 348 | |
| 286 | |
| 252 |
