- 3048 Views
- 3 replies
- 2 kudos
Resolved! Error with multiple FeatureLookup calls outside databricks
I access databricks feature store outside databricks with databricks-connect on my IDE pycharm.The problem is just outside Databricks, not with a notebook inside Databricks.I use FeatureLookup mecanism to pull data from Feature store tables in my cus...
- 3048 Views
- 3 replies
- 2 kudos
- 2 kudos
Also, Please refer to the below KB for additional resolution - https://learn.microsoft.com/en-us/azure/databricks/kb/dev-tools/dbconnect-protoserializer-stackoverflow
- 2 kudos
- 1136 Views
- 0 replies
- 0 kudos
How Delta Lake parses through the files on querying the table with a filter condition
Consider a table that gets partitioned on a date field. But, I'm filtering a column that is not partitioned. Now, with this filter condition whether all the files are parsed to attain the required result set, or does any data skipping happens?
- 1136 Views
- 0 replies
- 0 kudos
- 1546 Views
- 0 replies
- 3 kudos
Number of parquet files per delta table
Hello. We would like to understand how many parquet files are created per data table. To be more specific, we refer to the current snapshot of the table. For example, we noticed that while we performed initial inserts to a table, one parquet file was...
- 1546 Views
- 0 replies
- 3 kudos
- 4112 Views
- 4 replies
- 4 kudos
How to include visualizations returned from %run in the caller notebooks dashboard?
I have a notebook (nb1) that calls another one (nb2) via the %run command. This returns some visualizations that I want to add to a dashboard of the caller notebook (nb1-db). When I select the visualization drop down, then select Add to dashboard, th...
- 4112 Views
- 4 replies
- 4 kudos
- 4 kudos
Hi @Nicholas Couture​ , Could you please check and refer to https://docs.databricks.com/sql/user/security/access-control/dashboard-acl.html . Please let us know if this helps.
- 4 kudos
- 11198 Views
- 5 replies
- 17 kudos
Resolved! Azure AAD token with Databricks for User defined managed Identity inside Docker Container
Hi, We are currently using a Azure AAD Token inorder to authenticate with Databricks instead of generating Personal Access Tokens from Databricks. We have a multi-tenant architecture and so we are using Azure container instances to run multiple trans...
- 11198 Views
- 5 replies
- 17 kudos
- 17 kudos
Hi, This looks like an old issue which we have faced earlier, Could you please update the DBR to the latest version and try again?
- 17 kudos
- 25627 Views
- 9 replies
- 4 kudos
Referential Integrity (Primary Key / Foreign Key Constraint) - Azure Databricks SQL
Hello, Please suggest how can we implement Referential Integrity (Primary Key / Foreign Key Constraint) - between different tables defined on Azure Databricks Database. Basically the syntax to add Primary and Foreign Key constraint in the table defi...
- 25627 Views
- 9 replies
- 4 kudos
- 4 kudos
Is there any alternate way you could suggest to implement and enforce primary key constraint?
- 4 kudos
- 10081 Views
- 4 replies
- 4 kudos
Resolved! Issue with databricks and DRIVER_LIBRARY_INSTALLATION_FAILURE.?
I have like 5 maven libraries, and with all of them, I have the same issue with Job or Notebooks. How much do I have to wait? is there another solution? Thank you very much!
- 10081 Views
- 4 replies
- 4 kudos
- 4 kudos
@Raymond Garcia​ , could you please open a support case with Databricks for the same? We will triage the issue and provide a solution.
- 4 kudos
- 5253 Views
- 4 replies
- 0 kudos
Resolved! Can I use VMs from Pool for my Workflow cluster?
Hi,there is no option to take VMs from a Pool for a new workflow (Azure Cloud)?default schema for a new cluster:{ "num_workers": 0, "spark_version": "10.4.x-scala2.12", "spark_conf": { "spark.master": "local[*, 4]", "spark...
- 5253 Views
- 4 replies
- 0 kudos
- 0 kudos
@Michal Mlaka​ I just checked on the UI and I could find the pools listing under worker type in a job cluster configuration. It should work.
- 0 kudos
- 4540 Views
- 5 replies
- 2 kudos
Resolved! What will be the next LTS version after 10.4?
What will be the next LTS version after 10.4?
- 4540 Views
- 5 replies
- 2 kudos
- 2 kudos
Hello, 11.3 LTS is now available https://learn.microsoft.com/en-us/azure/databricks/release-notes/runtime/11.3
- 2 kudos
- 2512 Views
- 1 replies
- 2 kudos
How to access Delta Live Tables feature?
Hi, I tried following the Delta Live Tables quickstart (https://docs.databricks.com/data-engineering/delta-live-tables/delta-live-tables-quickstart.html), but I don't see the Pipelines tab under the Jobs page in my workspace. The same guide mentions...
- 2512 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi, you need a Premium workspace for the Pipelines tab to show up. This is what I see on my workspace with Standard Pricing Tier selected: And this is what what I see on my workspace with the Premium Pricing Tier:
- 2 kudos
- 4853 Views
- 3 replies
- 3 kudos
Using R, how do we write csv file to say dbfs:/tmp?
let us say I already have the data 'TotalData'write.csv(TotalData,file='/tmp/TotalData.csv',row.names = FALSE)I do not see any error from abovewhen I list files below:%fs ls /tmpI do not see any files written there. Why?
- 4853 Views
- 3 replies
- 3 kudos
- 3 kudos
Hi Thiam,Thank you for reaching out to us. In this case it seems that you have written a file to the OS /tmp and tried to fetch the same folder in DBFS.Written >> /tmp/TotalData.csvReading >> /dbfs/tmp/TotalData.csvPlease try to execute write.csv wit...
- 3 kudos
- 3760 Views
- 2 replies
- 3 kudos
Ingesting Kafka Avro into an Delta STREAMING LIVE TABLE
Using Azure Databricks:I can create a DLT table in python usingimport dlt import pyspark.sql.functions as fn from pyspark.sql.types import StringType @dlt.table( name = "<<landingTable>>", path = "<<storage path>>", comment = "<< descri...
- 3760 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi @John Mathews​ did you find a way to progress here?i am stuck in the same point...
- 3 kudos
- 1749 Views
- 1 replies
- 0 kudos
How to dedupe a source table prior to merge through JDBC SQL driver integration
Hi!We're working with change event data from relational and NoSQL databases then processing and ingesting that into DataBricks. It's streamed from source to our messaging platform. Then, our connector is pushing to DataBricks.Right now we're doing th...
- 1749 Views
- 1 replies
- 0 kudos
- 0 kudos
Update on the theory we are looking at. It'd be similar to below (with necessary changes to support best practices for MERGE such as reducing the search space):-- View for deduping pre-merge CREATE OR REPLACE TEMPORARY VIEW {view} AS SELECT * EXCEPT ...
- 0 kudos
- 2816 Views
- 3 replies
- 2 kudos
Resolved! Alert e-mail is not recognizing my html text
I've always used alert e-mail notifications with my custom message, written in HTML. The problem is that today it suddenly is not working anymore and I'm getting the alert e-mail notification distorted, as HTML doesn't work anymore.Does anyone know w...
- 2816 Views
- 3 replies
- 2 kudos
- 2 kudos
Apparently, it has been corrected and it is working again. Thank you everyone
- 2 kudos
- 10112 Views
- 4 replies
- 2 kudos
Resolved! Pandas API on Spark, Does it run on a multi-node cluster?
Hi, I have a few questions about "Pandas API on Spark". Thanks for your time to read my questions1) Input to these functions are Pandas DataFrame or PySpark DataFrame?2) When I use any pandas function (like isna, size, apply, where, etc ), does it ru...
- 10112 Views
- 4 replies
- 2 kudos
- 2 kudos
Hi @Mohammad Saber​ , Pandas dataset lives in the single machine, and is naturally iterable locally within the same machine. However, pandas-on-Spark dataset lives across multiple machines, and they are computed in a distributed manner. It is difficu...
- 2 kudos
| User | Count |
|---|---|
| 1619 | |
| 790 | |
| 483 | |
| 349 | |
| 287 |
