- 2125 Views
- 2 replies
- 0 kudos
How to exclude/skip a file temporarily in DLT
Hi,Is there any way to exclude a file from the dlt pipeline (autoload) run temporarily? What I mean is that I want to be able to exclude a specific file until I decided to include it in the load? I can't control the files or the location where they a...
- 2125 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi, I'm not aware of default Autoloader functionality that does what you're looking to do given that Autoloader is designed to incrementally ingest data as it arrives in cloud storage. Can you describe more about: "...exclude a specific file until I ...
- 0 kudos
- 1763 Views
- 1 replies
- 0 kudos
Regarding Databricks Terraform to new user
Hey FolksI am new to terraform and databricksI have usecase I want to create new user or add them to databricks workspace. And assign role to this user. And also assign cluster to this new userAfter 12hrs I want to delete this new user and also these...
- 1763 Views
- 1 replies
- 0 kudos
- 0 kudos
Thanks for this information
- 0 kudos
- 12132 Views
- 8 replies
- 4 kudos
Resolved! How to pass parameters in SSRS/Power BI (report builder) ?
Hello,In SSRS/Power BI (report builder), how to query a table in Databricks with parameters please ?Because this code doesn't works :SELECT * FROM TempBase.Customers WHERE Name = {{ @P_Name }}Thanks.
- 12132 Views
- 8 replies
- 4 kudos
- 4 kudos
Hi, I am not able to see the data in SSRS while I am using date parameters but with manual dates data is populating fine. The database is pointing to databricks. I am not sure what I am missing here. Please help me in this. ThanksI am trying with que...
- 4 kudos
- 2508 Views
- 2 replies
- 0 kudos
Unable to connect to oracle server from databricks notebook in AWS
I'm trying to connect to oracle server hosted in azure from AWS databricks notebook but seems the connection keeps timing out. I tested the connection IP using telnet <hostIP> 1521 command from another EC2 instance and that seems to reach the oracle ...
- 2508 Views
- 2 replies
- 0 kudos
- 0 kudos
@mh_db good day! Could you please confirm the Cluster type you used for testing? Was it a Shared Cluster, an Assigned/Single-User Cluster, or a No-Isolation cluster? Could you please try the same on the Assigned/Single User Cluster and No Isolation c...
- 0 kudos
- 2620 Views
- 3 replies
- 1 kudos
/api/2.0/preview/sql/queries API only returning certain queries
Hello,When using /api/2.0/preview/sql/queries to list out all available queries, I noticed that certain queries were being shown while others were not. I did a small test on my home workspace, and it was able to recognize certain queries when I defin...
- 2620 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi,How many queries were returned in the API call in question? The List Queries documentation describes this endpoint as supporting pagination with a default page size of 25, is that how many you saw returned? Query parameters page_size integer <= 10...
- 1 kudos
- 3070 Views
- 1 replies
- 1 kudos
spark/databricks temporary views and uuid
Hi All,We have a table which has an id column generated by uuid(). For ETL we use databricks/spark sql temporary views. we observed strange behavior between databricks sql temp view (create or replace temporary view) and spark sql temp view (df.creat...
- 3070 Views
- 1 replies
- 1 kudos
- 2003 Views
- 0 replies
- 0 kudos
Databricks Serverless Compute
Does Databricks support SQL Queries to extract data from Rest APIs?
- 2003 Views
- 0 replies
- 0 kudos
- 2279 Views
- 1 replies
- 1 kudos
Resolved! DataBricks Auto loader vs input source files deletion detection
Hi, While ingesting files from a source folder continuously, I would like to be able to detect the case where files are being deleted. As far as I can tell the Autoloader can not handle the detection of files deleted in the source folder. Hence the c...
- 2279 Views
- 1 replies
- 1 kudos
- 1 kudos
@Maatari Yes, it is true that Autoloader in Databricks cannot detect the deletion of files in the source folder during continuous ingestion. The Autoloader is designed to process files exactly once unless the option "cloudFiles.allowOverwrites" is en...
- 1 kudos
- 952 Views
- 0 replies
- 0 kudos
Datatype changed while writing in delta format
Hello team,I'm encountering an issue with my batch processing job. Initially, I write the job in overwrite mode with overwrite schema set to true. However, when I attempt to write the next batch in append mode, it fails due to a change in the datatyp...
- 952 Views
- 0 replies
- 0 kudos
- 2368 Views
- 2 replies
- 0 kudos
Lakehouse Federation Multi-User Authorization
Since Lakehouse Fed uses only one credential per connection to the foreign database, all queries using the connection will see all the data the credentials has to access to. Would anyone know if Lakehouse Fed will support authorization using the cred...
- 2368 Views
- 2 replies
- 0 kudos
- 0 kudos
@chardv, good day! Could you please share more details and the documentation [if you have referred any]?
- 0 kudos
- 13741 Views
- 1 replies
- 0 kudos
Azure Databricks Secret Management
Hi,Hope you both are doing well. I came to know that Databricks also provides secret management so I would like to compare it with some other well known secrets management solution like Azure Key Vault, CyberArk in the market . Can someone provides ...
- 13741 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi, A Databricks-backed secret scope is stored in an encrypted database managed by Azure Databricks. You can create a Databricks-backed secret scope using the Databricks CLI.The reason one should prefer the datarbricks secret manager is, You can easi...
- 0 kudos
- 16524 Views
- 8 replies
- 6 kudos
Databrick hive metastore location?
In databrick, where is hive metastore location is it control plane or data plane? for prod systems In terms of security what preventions should be taken to secure hive metastore?
- 16524 Views
- 8 replies
- 6 kudos
- 6 kudos
@as999​ The default metastore is managed by Databricks. If you are concerned about security and would like to have your own metastore you can go for the external metastore setup. You have the details steps in the below doc for setting up the external...
- 6 kudos
- 10669 Views
- 2 replies
- 2 kudos
Re-establish SparkSession using Databricks connect after cluster restart
Hello,when developing locally using Databricks connect how do I re-establish the SparkSession when the Cluster restarted? getOrCreate() seems to get the old invalid SparkSession even after Cluster restart instead of creating a new one or am I missing...
- 10669 Views
- 2 replies
- 2 kudos
- 2 kudos
If anyone encounters this problem, the solution that worked for me was to restart the Jupyter kernel.
- 2 kudos
- 1917 Views
- 1 replies
- 0 kudos
Unable to enforce schema on data read from jsonl file in Azure Databricks using pyspark
I'm tring to build a ETL pipeline in which I'm reading the jsonl files from the azure blob storage, then trying to transform and load it to delta tables in databricks. I have created the below schema for loading my data : schema = StructType([ S...
- 1917 Views
- 1 replies
- 0 kudos
- 0 kudos
Try this.Add option("multiline","true")
- 0 kudos
- 1975 Views
- 2 replies
- 1 kudos
upload file/table to delta table using CLI
Hi,I am using CLI to transfer local files to Databricks Volume. At the end of my upload, I want to create a meta table (storing file name, location, and some other information) and have it as a table on databricks Volume. I am not sure how to create ...
- 1975 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @pshuk , Greetings! We understand that you are looking for a CLI command to create a Table but at this moment Databricks doesn't support CLI command to create the table but you can use SQL Execution API -https://docs.databricks.com/api/workspace/...
- 1 kudos
| User | Count |
|---|---|
| 1624 | |
| 790 | |
| 506 | |
| 349 | |
| 287 |
