- 1892 Views
- 1 replies
- 3 kudos
Connecting Power BI on Azure to Databricks on AWS?
Can someone share with me the proper way to connect Power BI running on Azure to Databricks running on AWS?
- 1892 Views
- 1 replies
- 3 kudos
- 3 kudos
Have the same question. Do we have to take care of any specific tasks to make it work. Anyone who implemented it?
- 3 kudos
- 5353 Views
- 1 replies
- 0 kudos
How to import excel files xls/xlsx file into Databricks python notebook?
Method 1: Using "com.crealytics.spark.excel" package, how do I import the package?Method 2: Using pandas I tried the possible paths, but file not found it shows, nor while uploading the xls/xlsx file it shows options for importing the dataframe.Help ...
- 5353 Views
- 1 replies
- 0 kudos
- 0 kudos
import pandas as pd ExcelData = pd.read_excel("/dbfs"+FilePath, sheetName) # make sure you add /dbfs to FilePath
- 0 kudos
- 2132 Views
- 0 replies
- 0 kudos
Error accessing Managed Table with Row Level Security using Databricks Cluster
Hi Everyone,We are trying to implement Row Level Security In Delta Table and done testing (i.e. sql execution api,sql editor,sql notebook) using Sql Serverless in Unity Catalog.But when tried to access the table having RLS in Notebook using Pyspark w...
- 2132 Views
- 0 replies
- 0 kudos
- 2820 Views
- 1 replies
- 0 kudos
Bug in unity catalog registering external database that is case sensitive
I successfully registered in my Unity Catalog an external Database ```dwcore``` that is hosted on SQL server.I first added the connection in "External Data": tested the connection and it was successful.I then added the database on top: tested the con...
- 2820 Views
- 1 replies
- 0 kudos
- 1463 Views
- 0 replies
- 0 kudos
Dask-Databricks init script not working
Hello everybody ! I am trying to use the Dask-Databricks distribution (https://github.com/dask-contrib/dask-databricks?tab=readme-ov-file)i set up the required init-script according to the instructions on the Github page and had no problems there, h...
- 1463 Views
- 0 replies
- 0 kudos
- 12827 Views
- 7 replies
- 2 kudos
- 12827 Views
- 7 replies
- 2 kudos
- 2 kudos
Hi @amitca71 @atanu .. yes you can associate as many vpcs(workspace deployment fundamental) across regions and aws accounts to one single databricks aws account infact its one of the super powers of databricks platform and you can even track all thei...
- 2 kudos
- 1531 Views
- 0 replies
- 0 kudos
SELECT issue after an OPTIMIZE operation
I have a strange issue after an OPTIMIZE, no results are returned anymore.I can time travel over the version easily but passed this data nothing when I'm doing a simple SELECT *.But I still got a result when I'm doing a SELECT count(*).How is this po...
- 1531 Views
- 0 replies
- 0 kudos
- 18088 Views
- 16 replies
- 20 kudos
Resolved! network security for DBFS storage account
In Azure Databricks the DBFS storage account is open to all networks. Changing that to use a private endpoint or minimizing access to selected networks is not allowed.Is there any way to add network security to this storage account? Alternatively, is...
- 18088 Views
- 16 replies
- 20 kudos
- 20 kudos
How can we secure the storage account in the managed resource group which holds the DBFS with restricted network access, since access from all networks is blocked by our Azure storage account policy?
- 20 kudos
- 2004 Views
- 0 replies
- 0 kudos
Parse_Syntax_Error Help
i'm getting this error: Exception in thread "main" org.apache.spark.sql.catalyst.parser.ParseException: [PARSE_SYNTAX_ERROR] Syntax error at or near ','.(line 1, pos 18) == SQL == sum(mp4) AS Videos, sum(csv+xlsx) AS Sheets, sum(docx+txt+pdf) AS Docu...
- 2004 Views
- 0 replies
- 0 kudos
- 10185 Views
- 6 replies
- 2 kudos
Resolved! How to grant access to views without granting access to underlying tables
I have a medallion architecture: Bronze layer: Raw data in tablesSilver layer: Refined data in views created from the bronze layerGold layer: Data products as views created from the silver layerCurrently I have a data scientist that needs access to d...
- 10185 Views
- 6 replies
- 2 kudos
- 2 kudos
Single-user clusters use a different security mode which is the reason for this difference. On single-user/assigned clusters, you'll need the Fine Grained Access Control service (which is a Serverless service) - that is the solution to this problem (...
- 2 kudos
- 5201 Views
- 3 replies
- 0 kudos
Delta live tables straming
I'm trying to addmonotonicallyIncreasingId() column to a streaming table and I see the following errorFailed to start stream [table_name] in either append mode or complete mode. Append mode error: Expression(s): monotonically_increasing_id() is not s...
- 5201 Views
- 3 replies
- 0 kudos
- 0 kudos
Is aggregations with row_number() combined with a SQL window function and a watermark still supported in Databricks 14.3?
- 0 kudos
- 6405 Views
- 5 replies
- 0 kudos
Is there a way to control the cluster runtime version for DLT
Hi team, When I create a DLT job, is there a way to control the cluster runtime version somewhere? E.g. I want to use 14.3 LTS. I tried to add `"spark_version": "14.3.x-scala2.12",` inside cluster default label but not work.Thanks
- 6405 Views
- 5 replies
- 0 kudos
- 0 kudos
Thanks. Got it.And the cluster has to be share mode. Can different DLT jobs share clusters or when DLT job is running, can other people use the cluster? Seems each DLT job running will start a new cluster. If it is not be able to shared, why it has t...
- 0 kudos
- 2160 Views
- 1 replies
- 0 kudos
pyspark.pandas PandasNotImplementedError
Can someone explain why this below code is throwing an error? My intuition is telling me it's my spark version (3.2.1) but would like confirmation:d = {'key':['a','a','c','d','e','f','g','h'], 'data':[1,2,3,4,5,6,7,8]} x = ps.DataFrame(d) x[x['...
- 2160 Views
- 1 replies
- 0 kudos
- 0 kudos
@pjp94 - The error indicates the pandas pyspark implementation does not have the below method implemented. pd.Series.duplicated() Next steps is to use dataframe methods such as distinct, groupBy, dropDuplicates to resolve this.
- 0 kudos
- 2904 Views
- 1 replies
- 0 kudos
TimeoutException: Stream Execution thread for stream [xxxxxx]failed to stop within 15000 millisecond
TimeoutException: Stream Execution thread for stream [id = xxx runId = xxxx] failed to stop within 15000 milliseconds (specified by spark.sql.streaming.stopTimeout). See the cause on what was being executed in the streaming query thread.I have a data...
- 2904 Views
- 1 replies
- 0 kudos
- 0 kudos
@User_1611 - could you please try the following ? Reduce the number of streaming queries running on the same clusterMake sure your code does not try to re-trigger/start an active streaming queryMake sure to collect the thread dumps if this error hap...
- 0 kudos
- 6833 Views
- 5 replies
- 0 kudos
Read large volume of parquet files
I have 50k + parquet files in the in azure datalake and i have mount point as well. I need to read all the files and load into a dataframe. i have around 2 billion records in total and all the files are not having all the columns, column order may di...
- 6833 Views
- 5 replies
- 0 kudos
- 0 kudos
@Shan1 - This could be due to the files have cols that differ by data type. Eg. Integer vs long , Boolean vs integer. can be resolved by schemaMerge=False. Please refer to this code. https://github.com/apache/spark/blob/418bba5ad6053449a141f3c9c31e...
- 0 kudos
| User | Count |
|---|---|
| 1629 | |
| 790 | |
| 511 | |
| 349 | |
| 287 |
