- 2832 Views
- 2 replies
- 0 kudos
Upload to Volume
How to programmatically upload parquet files from Azure data lake to Catalog's Volumes? source_path = "abfss://datalake-raw-dev@xxx.dfs.core.windows.net/xxxxx/saxxles/xx/source/ETL/transformed_data/parquet/" # Define the path to your Unity Catalog V...
- 2832 Views
- 2 replies
- 0 kudos
- 0 kudos
Besides, when accessing volumes, you don't need to provide dbfs protocol: `/Volumes/xxx/xxx/transformed_parquet`
- 0 kudos
- 785 Views
- 1 replies
- 0 kudos
helps on receiving events with python
According to the python code of Create a Python script to receive events, I need to define BLOB_STORAGE_CONNECTION_STRING and BLOB_CONTAINER_NAME, Could you tell me how to get them? Thanks.
- 785 Views
- 1 replies
- 0 kudos
- 0 kudos
Hey @lixing How is this actually related to Databricks?Besides that, these are just meta information of your storage container. One way to get these data is to go to the Azure portal, and navigate to the appropriate container.
- 0 kudos
- 3956 Views
- 2 replies
- 2 kudos
Resolved! How to run OPTIMIZE to too big data set which has 11TB and more ?
Sorry for my very poor English and low Databricks Skill.At work, my boss asked me to perform liquid clustering on four columns for a Delta Lake table with an 11TB capacity and over 80 columns, and I was estimating the resources and costs required to ...
- 3956 Views
- 2 replies
- 2 kudos
- 2 kudos
Couple of things:OPTIMIZE is a very compute intensive operation. Make sure you pick a VM that is compute optimized.I had to look into the AWS instances but it seems the r6g.large you're using is just a 2 CPU 16GB machine. This is by far not sufficien...
- 2 kudos
- 1554 Views
- 0 replies
- 0 kudos
Delta Live Table pipeline hanging at INITIALIZING forever
I have a dlt pipeline haning at INIALIZING forever, it never stops. But I found the Analysis Exeption already happened at beginningpyspark.errors.exceptions.captured.AnalysisException: [UNRESOLVED_COLUMN.WITH_SUGGESTION] A column, variable, or functi...
- 1554 Views
- 0 replies
- 0 kudos
- 1887 Views
- 0 replies
- 0 kudos
"No module named google.cloud.spark" errors querying BigQuery
Personal Cluster 15.3 ML, Running the following notebook:import pyspark.sql.functions as F from datetime import datetime, timedelta spark.sparkContext.addPyFile("gs://spark-lib/bigquery/spark-bigquery-support-0.26.0.zip") target_hour = datetime(202...
- 1887 Views
- 0 replies
- 0 kudos
- 3009 Views
- 2 replies
- 0 kudos
Delta Live Tables and Pivoting
Hello,I'm trying to create a DLT pipeline where I read data as a streaming dataset from a Kafka source, save it in a table, and then filter, transform, and pivot the data. However, I've encountered an issue: DLT doesn't support pivoting, and using fo...
- 3009 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @YS1 ,As a workaround you can rewrite pivot to sql with case statements.Below Pivot:data = [ ("ProductA", "North", 100), ("ProductA", "South", 150), ("ProductA", "East", 200), ("ProductA", "West", 250), ("ProductB", "North", 30...
- 0 kudos
- 7464 Views
- 6 replies
- 4 kudos
OAuth U2M Manual token generation failing
I am writing a frontend webpage that will log into DataBricks and allow the user to select datasets.I am new to front end development, so there may be some things I am missing here, but I know that the DataBricks SQL connector for javascript only wor...
- 7464 Views
- 6 replies
- 4 kudos
- 4 kudos
@szymon_dybczak Thank you for your help. Your solution has worked very well for me.
- 4 kudos
- 9455 Views
- 10 replies
- 5 kudos
Jobs API 2.1 OpenAPI specification seems broken.
Jobs API 2.1 OpenAPI specification seems broken.The swagger file seems to be invalid.https://docs.databricks.com/_extras/api-refs/jobs-2.1-aws.yaml
- 9455 Views
- 10 replies
- 5 kudos
- 5 kudos
@Debayan Mukherjee​ , are you suggesting to revert the openapi version specified in https://docs.databricks.com/_extras/api-refs/jobs-2.1-aws.yaml from 3.1.0 to 3.0.3?
- 5 kudos
- 1721 Views
- 2 replies
- 2 kudos
Keywords and Functions supported in SQL but not in Databricks SQL.
Actually, I have around 2000 SQL queries. I have to convert them in Databricks supported SQLs, so that I can run them in databricks environment. So I want to know the list of all keywords, functions or anything that is different in databricks SQL. Pl...
- 1721 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @RishabhGarg ,You're saying SQL, but which dialect? Because every provider has its own extension to ANSI SQL standard. So for example, if you're using SQL Server for example, there is a TOP keyword to limit the rows.
- 2 kudos
- 3576 Views
- 6 replies
- 3 kudos
Resolved! Databricks Variant Data Type
HiI've tried to enabled a table to test the new variant data type (public preview)I used the alter cmd: ALTER TABLE tablexxxx SET TBLPROPERTIES('delta.feature.variantType-preview' = 'supported')and I have the error[DELTA_UNSUPPORTED_FEATURES_IN_CONFI...
- 3576 Views
- 6 replies
- 3 kudos
- 3 kudos
Yes but I tried using 2 cluster typesSQL Serverless and New Compute Serverless.Error in both
- 3 kudos
- 4002 Views
- 3 replies
- 1 kudos
Attributed-Based Access Control
Hi,Over here they are explaining attribute-based-access-controls, which I want to implement in my project but can't find the documentation or the option to create rules myself. Is this feature already available?https://www.databricks.com/dataaisummit...
- 4002 Views
- 3 replies
- 1 kudos
- 719 Views
- 0 replies
- 1 kudos
Include SPARK-46990 in databricks 15.4 LTS
Hi, could you include fix SPARK-46990 ([SPARK-46990] Regression: Unable to load empty avro files emitted by event-hubs - ASF JIRA (apache.org)) in Databricks 15.4? (15.4 is in the beta stage, so it might be a right time to include fix)
- 719 Views
- 0 replies
- 1 kudos
- 1826 Views
- 1 replies
- 0 kudos
Resolved! Compute Policy Does Not Install Libraries
Has anyone else run into the issue where applying libraries through a compute policy just completely does not work?I'm trying to insane some pretty basic Python libraries from PyPI (pytest and paramiko, for example) and it is failing on 13.3 and 14.3...
- 1826 Views
- 1 replies
- 0 kudos
- 3481 Views
- 2 replies
- 0 kudos
Incrementally ingesting from a static db into a Delta Table
Hello everyone,I’m very new to Delta Live Tables (and Delta Tables too), so please forgive me if this question has been asked here before.Some context: I have over 100M records stored in a Postgres table. I can connect to this table using the convent...
- 3481 Views
- 2 replies
- 0 kudos
- 0 kudos
If I'm reading this right you created a materialized view to prep your data in Postgres. You may not need to do that, and it will also limit your integration options. It puts more work on Postgres, usually creates more data to move, and will not as m...
- 0 kudos
- 3313 Views
- 3 replies
- 1 kudos
Not able to unzip the zip file with mount and unity catalog
Hello Team, I have a zip file in ADLS Gen 2. The folder I am using is mounted and when I run command : dbutils.fs.ls(path) it lists all the files(including the zip require). However, when I try to read the zip using 'zipfile' module, it displays 'Fil...
- 3313 Views
- 3 replies
- 1 kudos
- 1 kudos
@Gareema, since you're using UC, can you use Volumes instead? It basically replaces the old mount approach.
- 1 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 566 | |
| 349 | |
| 287 |
