- 3924 Views
- 6 replies
- 0 kudos
PowerBI Connection: Possible to use ADOMDClient (or alternative)?
I wish to extract from PowerBI Datasets the metadata of all Measures, Relationships and Entities.In VSCode I have a python script that connects to the PowerBI API using the Pyadomd module connecting via the XMLA endpoint. After much trial and error I...
- 3924 Views
- 6 replies
- 0 kudos
- 0 kudos
you've any luck? i have same requirement, wanna read some datasets from powerbi datamodel to my databricks workspace and store in datalake.
- 0 kudos
- 1980 Views
- 3 replies
- 1 kudos
Resolved! Databricks App - Streamlit file Upload issue
I used this code snippet from cookbook and created a custom databricks streamlit app to upload files to the volume but i am getting the following error even before start uploading it to volume. Using file_uploader in streamlit while uploading the fil...
- 1980 Views
- 3 replies
- 1 kudos
- 4792 Views
- 4 replies
- 2 kudos
Move managed DLT table from one schema to another schema in Databricks
I have a DLT table in schema A which is being loaded by DLT pipeline.I want to move the table from schema A to schema B, and repoint my existing DLT pipeline to table in schema B. also I need to avoid full reload in DLT pipeline on table in Schema B....
- 4792 Views
- 4 replies
- 2 kudos
- 2 kudos
Have you tried the belowPause or Stop the DLT PipelinePrevent new writes while moving the table.2.Move the Table in Metastore DLT uses Delta tables under the hood, so you can move the table in the metastore without copying data:ALTER TABLE schemaA.ta...
- 2 kudos
- 17836 Views
- 12 replies
- 7 kudos
Pyspark: You cannot use dbutils within a spark job
I am trying to parallelise the execution of file copy in Databricks. Making use of multiple executors is one way. So, this is the piece of code that I wrote in pyspark.def parallel_copy_execution(src_path: str, target_path: str): files_in_path = db...
- 17836 Views
- 12 replies
- 7 kudos
- 7 kudos
If you have spark session, you can use Spark hidden File System:# Get FileSystem from SparkSession fs = spark._jvm.org.apache.hadoop.fs.FileSystem.get(spark._jsc.hadoopConfiguration()) # Get Path class to convert string path to FS path path = spark._...
- 7 kudos
- 1114 Views
- 3 replies
- 1 kudos
Resolved! How to define a column tag in a table schema definition?
Setting a tag for a specific column can be done using the SQL command:https://docs.databricks.com/aws/en/sql/language-manual/sql-ref-syntax-ddl-set-tagIs there another possible way to define a column tag?For example, it is possible to add a column co...
- 1114 Views
- 3 replies
- 1 kudos
- 1 kudos
You’re welcome @b-baran ! If you feel my answer addressed your question, could you please mark it as the solution to the post? Thank you very much!
- 1 kudos
- 384 Views
- 1 replies
- 1 kudos
Is CREATE STREAMING LIVE VIEW deprecated?
Hi,I'm trying to learn Lakeflow Pipelines (DLT) and found some examples online where the CREATE STREAMING LIVE VIEW statement is used. When I try to search for it in the Databricks document, there is nothing really I can find on them.https://docs.dat...
- 384 Views
- 1 replies
- 1 kudos
- 1 kudos
Hello @GeertR ,The LIVE virtual schema is a legacy feature of Lakeflow Declarative Pipelines (DLT) and is deprecated. You can still use it with pipelines that were created in legacy publishing mode, but the pipeline configuration UI no longer lets yo...
- 1 kudos
- 2368 Views
- 2 replies
- 0 kudos
importing files from streamlit app on databricks to dbfs
I am building a Streamlit-based app on Databricks that allows users to:Upload Excel scenario filesStore them in DBFS (e.g.,/FileStore/SCO/scenarios/)Trigger a simulation/optimization model using the uploaded/stored file as input to the model Store th...
- 2368 Views
- 2 replies
- 0 kudos
- 0 kudos
Here is an example for uploading files to a Volume, a download example is there, too
- 0 kudos
- 16513 Views
- 4 replies
- 4 kudos
Resolved! UUID/GUID Datatype in Databricks SQL
Hi all,I am trying to create a table with a GUID column.I have tried using GUID, UUID; but both of them are not working.Can someone help me with the syntax for adding a GUID column?Thanks!
- 16513 Views
- 4 replies
- 4 kudos
- 4 kudos
> What ANSI SQL data structure to use for UUID or GUID?I had similar question. The answer was `STRING`.
- 4 kudos
- 16115 Views
- 4 replies
- 1 kudos
Cluster in Pending State for long time
Pending for a long time at this stage “Finding instances for new nodes, acquiring more instances if necessary”. How can this be fixed?
- 16115 Views
- 4 replies
- 1 kudos
- 1 kudos
I faced similar situation yesterday. So I kept waiting instead of locking my system or closing the tabs. After sometime it went all fine.
- 1 kudos
- 1688 Views
- 2 replies
- 1 kudos
How to Use BladeBridge for Redshift to Databricks Migration?
Hi all,I have a Redshift queries that I need to migrate to Databricks using BladeBridge, but I have never used BladeBridge before and can’t find any clear documentation or steps on how to use it within the Databricks environment.If anyone has already...
- 1688 Views
- 2 replies
- 1 kudos
- 1 kudos
Dear @lingareddy_Alva ,Thank you so much for sharing these steps & specifics. Much appreciated!Context:Have just started exploring BladeBridge for AWS Redshift to Databricks migration. "BladeBridge operates as a code translation framework" and it sup...
- 1 kudos
- 3829 Views
- 1 replies
- 0 kudos
How to allow Table deletion without requiring ownership on table? Problem Description In DBR 6 (and earlier), a non-admin user can delete a table that...
How to allow Table deletion without requiring ownership on table?Problem DescriptionIn DBR 6 (and earlier), a non-admin user can delete a table that the user doesn't own, as long as the user has ownership on the table's parent database (perhaps throu...
- 3829 Views
- 1 replies
- 0 kudos
- 0 kudos
I am having the same issue but on Python 3.10.12.I need to be able to have another user have "manage" access to a table in the unity catalog. We both have write access to the schema.
- 0 kudos
- 995 Views
- 1 replies
- 0 kudos
DBT task status update gets delayed for several minutes
Hi, our team has recently begun experiencing a several-minute delay between Databricks DBT tasks finishing the computations and the subsequent status update from running state to success. The DBT project is part of the workspace git repo. In the fir...
- 995 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi Rosty,How are you doing today? thanks for sharing the detailed context. I agree, it definitely sounds frustrating to have DBT tasks showing delays even after finishing the actual work. Based on what you've described, the delay is likely happening ...
- 0 kudos
- 391 Views
- 1 replies
- 0 kudos
Move large SQL data into Databricks
Hello, I have a large on-prem SQL database (~15TB). It heavily utilizes the sql_variant datatype. Would like to move it into a Databricks bronze layer, and have it synchronized as close to 'live' as possible. What could be the solution? It seems like...
- 391 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @zychoo ,I would consider a “Near-real-time” solution into Databricks Bronze, something like:- Log-based CDC tool (Qlik / Debezium / HVR) captures changes from SQL Server.- Tool serializes sql_variant to JSON or string+type metadata.- Writes to S3...
- 0 kudos
- 1727 Views
- 1 replies
- 1 kudos
AWS Databricks and Fabric OneLake integration
Bit of a weird scenario and I wanted to hear from the experts in this communityLets say I have a Fabric Lakehouse (OneLake) and I want to read that data into Databricks (AWS) Unity Catalog to play with that data. What is the recommended mechanism to ...
- 1727 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @de2298, Currently, Microsoft Fabric does not offer a built-in connector that allows direct querying or exposure of Delta Share tables from AWS Databricks into a Fabric Warehouse. The Unity Catalog mirroring feature is supported only with Azure Da...
- 1 kudos
- 1821 Views
- 4 replies
- 0 kudos
Resolved! Declarative Pipelines with datacontracts
I'm wondering if anyone has successfully integrated data contracts with declarative pipelines in Databricks. Specifically, I want to reuse the quality checks and schema definitions from the contract directly within the pipeline's stages. I haven't fo...
- 1821 Views
- 4 replies
- 0 kudos
- 0 kudos
Suggested Steps:Define the data contractCreate a YAML/JSON file containing:Schema (column names, data types, required fields)Data quality rules (null checks, ranges, regex patterns, allowed value lists)Governance metadata (e.g., data sensitivity, LGP...
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 791 | |
| 553 | |
| 349 | |
| 287 |
