- 10482 Views
- 4 replies
- 2 kudos
Resolved! Problem when updating Databricks Repo through DevOps Pipeline
Hello all!I've been working on integrating a Databricks Repos update API call to a DevOps Pipeline so that the Databricks local repo stays up to date with the remote staging branch (Pipeline executes whenever there's a new commit in to the staging br...
- 10482 Views
- 4 replies
- 2 kudos
- 2 kudos
@BookerE1 I found it!. There was already another thread related to this problem and someone else helped me find the solution (Problem was the pool that I was using for the pipeline)This is the link to the other thread: https://community.databricks.co...
- 2 kudos
- 2215 Views
- 2 replies
- 0 kudos
Pipeline API documentation issue
In List Pipelines API documentation, at response is specified that response is statuses Array of objects The list of events matching the request criteria. next_page_token string If present, a token to fetch the next page of events. But if we retri...
- 2215 Views
- 2 replies
- 0 kudos
- 0 kudos
@costi9992 This has been documented now: https://docs.databricks.com/api/workspace/jobs/listruns
- 0 kudos
- 22176 Views
- 0 replies
- 0 kudos
What are optimized solutions for moving on-premise Hadoop data
Hi Team ,What are optimized solutions for moving on-premise Hadoop/hadoop distributed file system parquet data to Databricks as Delta file?Regards,Phanindra
- 22176 Views
- 0 replies
- 0 kudos
- 1931 Views
- 1 replies
- 3 kudos
Connecting Power BI on Azure to Databricks on AWS?
Can someone share with me the proper way to connect Power BI running on Azure to Databricks running on AWS?
- 1931 Views
- 1 replies
- 3 kudos
- 3 kudos
Have the same question. Do we have to take care of any specific tasks to make it work. Anyone who implemented it?
- 3 kudos
- 5497 Views
- 1 replies
- 0 kudos
How to import excel files xls/xlsx file into Databricks python notebook?
Method 1: Using "com.crealytics.spark.excel" package, how do I import the package?Method 2: Using pandas I tried the possible paths, but file not found it shows, nor while uploading the xls/xlsx file it shows options for importing the dataframe.Help ...
- 5497 Views
- 1 replies
- 0 kudos
- 0 kudos
import pandas as pd ExcelData = pd.read_excel("/dbfs"+FilePath, sheetName) # make sure you add /dbfs to FilePath
- 0 kudos
- 2183 Views
- 0 replies
- 0 kudos
Error accessing Managed Table with Row Level Security using Databricks Cluster
Hi Everyone,We are trying to implement Row Level Security In Delta Table and done testing (i.e. sql execution api,sql editor,sql notebook) using Sql Serverless in Unity Catalog.But when tried to access the table having RLS in Notebook using Pyspark w...
- 2183 Views
- 0 replies
- 0 kudos
- 2929 Views
- 1 replies
- 0 kudos
Bug in unity catalog registering external database that is case sensitive
I successfully registered in my Unity Catalog an external Database ```dwcore``` that is hosted on SQL server.I first added the connection in "External Data": tested the connection and it was successful.I then added the database on top: tested the con...
- 2929 Views
- 1 replies
- 0 kudos
- 1515 Views
- 0 replies
- 0 kudos
Dask-Databricks init script not working
Hello everybody ! I am trying to use the Dask-Databricks distribution (https://github.com/dask-contrib/dask-databricks?tab=readme-ov-file)i set up the required init-script according to the instructions on the Github page and had no problems there, h...
- 1515 Views
- 0 replies
- 0 kudos
- 13242 Views
- 7 replies
- 2 kudos
- 13242 Views
- 7 replies
- 2 kudos
- 2 kudos
Hi @amitca71 @atanu .. yes you can associate as many vpcs(workspace deployment fundamental) across regions and aws accounts to one single databricks aws account infact its one of the super powers of databricks platform and you can even track all thei...
- 2 kudos
- 1560 Views
- 0 replies
- 0 kudos
SELECT issue after an OPTIMIZE operation
I have a strange issue after an OPTIMIZE, no results are returned anymore.I can time travel over the version easily but passed this data nothing when I'm doing a simple SELECT *.But I still got a result when I'm doing a SELECT count(*).How is this po...
- 1560 Views
- 0 replies
- 0 kudos
- 18602 Views
- 16 replies
- 20 kudos
Resolved! network security for DBFS storage account
In Azure Databricks the DBFS storage account is open to all networks. Changing that to use a private endpoint or minimizing access to selected networks is not allowed.Is there any way to add network security to this storage account? Alternatively, is...
- 18602 Views
- 16 replies
- 20 kudos
- 20 kudos
How can we secure the storage account in the managed resource group which holds the DBFS with restricted network access, since access from all networks is blocked by our Azure storage account policy?
- 20 kudos
- 2063 Views
- 0 replies
- 0 kudos
Parse_Syntax_Error Help
i'm getting this error: Exception in thread "main" org.apache.spark.sql.catalyst.parser.ParseException: [PARSE_SYNTAX_ERROR] Syntax error at or near ','.(line 1, pos 18) == SQL == sum(mp4) AS Videos, sum(csv+xlsx) AS Sheets, sum(docx+txt+pdf) AS Docu...
- 2063 Views
- 0 replies
- 0 kudos
- 10431 Views
- 6 replies
- 2 kudos
Resolved! How to grant access to views without granting access to underlying tables
I have a medallion architecture: Bronze layer: Raw data in tablesSilver layer: Refined data in views created from the bronze layerGold layer: Data products as views created from the silver layerCurrently I have a data scientist that needs access to d...
- 10431 Views
- 6 replies
- 2 kudos
- 2 kudos
Single-user clusters use a different security mode which is the reason for this difference. On single-user/assigned clusters, you'll need the Fine Grained Access Control service (which is a Serverless service) - that is the solution to this problem (...
- 2 kudos
- 5361 Views
- 3 replies
- 0 kudos
Delta live tables straming
I'm trying to addmonotonicallyIncreasingId() column to a streaming table and I see the following errorFailed to start stream [table_name] in either append mode or complete mode. Append mode error: Expression(s): monotonically_increasing_id() is not s...
- 5361 Views
- 3 replies
- 0 kudos
- 0 kudos
Is aggregations with row_number() combined with a SQL window function and a watermark still supported in Databricks 14.3?
- 0 kudos
- 6740 Views
- 5 replies
- 0 kudos
Is there a way to control the cluster runtime version for DLT
Hi team, When I create a DLT job, is there a way to control the cluster runtime version somewhere? E.g. I want to use 14.3 LTS. I tried to add `"spark_version": "14.3.x-scala2.12",` inside cluster default label but not work.Thanks
- 6740 Views
- 5 replies
- 0 kudos
- 0 kudos
Thanks. Got it.And the cluster has to be share mode. Can different DLT jobs share clusters or when DLT job is running, can other people use the cluster? Seems each DLT job running will start a new cluster. If it is not be able to shared, why it has t...
- 0 kudos
| User | Count |
|---|---|
| 1637 | |
| 791 | |
| 534 | |
| 349 | |
| 287 |
