- 2656 Views
- 2 replies
- 0 kudos
Records are missing while creating new data from one big dataframe using filter
Hi,I have data in file like below I have different types of row in my input file, column number 8 defines the type of the record.In the above file we have 4 types of records 00 to 03My requirement is:There will be multiple files in the source path, e...
- 2656 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Retired_mod ,If i run again with same files sometimes records will be missed from same files of the previous run or records will be missed from different file.Example:run1: 1 record missing in file1, no issue with other filesrun2: 1 record missin...
- 0 kudos
- 3682 Views
- 0 replies
- 0 kudos
Dashboard backup/download
Hola all, I'm trying to download all the dashboard definitions, however, I can only download the folder structure with no file inside. The procedure I'm using is* Go to the dashboard folder* Download it as DBC or source archiveUnfortunately, the DBC ...
- 3682 Views
- 0 replies
- 0 kudos
- 1261 Views
- 1 replies
- 0 kudos
ganglia metrics
Hello Everyone, I have build this script in order to collect ganglia metrics but the size of stderr and sdtout ganglia is 0. It doesn't work. I Have put this script on Workspace due to migration databricks all init-script should be place on Workspace...
- 1261 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi, Is there any error you are getting? Also, please tag @Debayan with your next comment which will notify me. Thanks!
- 0 kudos
- 1620 Views
- 1 replies
- 0 kudos
GCP hosted Databricks - DBFS temp files - Not Found
I've been working on obtaining DDL at the schema level in Hive Metastore within GCP-hosted Databricks. I've implemented a Python code that generates SQL files in the dbfs/temp directory. However, when running the code, I'm encountering a "file path n...
- 1620 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi, the error code snippet with the whole error may help to determine the issue, also, considering the above points may also work as a fix.
- 0 kudos
- 2614 Views
- 1 replies
- 0 kudos
Unable to find permission button in Sql Warehouse for providing Access
Hi Everyone, am unable to see the permission button in sql warehouse to provide access to other users.I have admin rights and databricks is premium subscription.
- 2614 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi, Could you please provide a screenshot of the SQL warehouse? Also, you can go through: https://docs.databricks.com/en/security/auth-authz/access-control/sql-endpoint-acl.htmlAlso, please tag @Debayan with your next comment which will notify me. Th...
- 0 kudos
- 1156 Views
- 1 replies
- 1 kudos
SQL Serverless - cost view
Hi,Anyone knows how I'm able to monitor cost of the SQL Serverless? I'm using Databricks in Azure and I'm not sure where to find cost generated by compute resources hosted on Databricks.
- 1156 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi, You can calculate the pricing in https://www.databricks.com/product/pricing/databricks-sql also, https://azure.microsoft.com/en-in/pricing/details/databricks/#:~:text=Sign%20in%20to%20the%20Azure,asked%20questions%20about%20Azure%20pricing. For A...
- 1 kudos
- 3097 Views
- 2 replies
- 0 kudos
Error while creating external table in unity catalog
I am trying to create an external table using csv file which is stored in ADLS gen2 My account owner has created a storage credential and an external location I am a databricks user who all privileges on external location when trying to create a tabl...
- 3097 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Shubhanshu , Could you please try https://learn.microsoft.com/en-us/answers/questions/1314651/error-invalid-configuration-value-detected-for-fs and see if this is working?
- 0 kudos
- 2112 Views
- 1 replies
- 0 kudos
Issue with Databricks notebooks and 'requests' python library. Incosistent output.
I have a strange issue with Databricks notebooks and Google Colab notebooks, where I cannot get the results from requests library that would be consistent to what I get on my local computer. Dir Surveys (wyo.gov) ("http://pipeline.wyo.gov/r_Direction...
- 2112 Views
- 1 replies
- 0 kudos
- 0 kudos
Additional information: when I download html file from the above mentioned website, I still get not the same html page as I get from the same code that I run on my computer or any other computer. The size of downloaded html file is the same as I get ...
- 0 kudos
- 2336 Views
- 1 replies
- 0 kudos
REST API
Creating an application to capture cluster metrics and sending HTTP REST request to the Spark History Server's API endpoint to retrieve a list of applications. This request doesn't generate logs in the Spark History Server's log files. The Spark Hist...
- 2336 Views
- 1 replies
- 0 kudos
- 3142 Views
- 1 replies
- 0 kudos
Monitoring job metrics
Hi,We need to monitor Databricks jobs and we have made a setup where are able to get the prometheus metrics, however, we are lagging an overview of which metrics refer to what.Namely, we need to monitor the following:failed jobs : is a job failedtabl...
- 3142 Views
- 1 replies
- 0 kudos
- 0 kudos
I have reposted this post in "Administation and Architecture"Monitoring job metrics - Databricks - 42956
- 0 kudos
- 921 Views
- 0 replies
- 0 kudos
ListBucket
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:ListBucket" ], "Resource": [ "arn:aws:s3:::<s3-bucket-name>" ] }, { "Effect": "Allow", "Action": [ "s3:Pu...
- 921 Views
- 0 replies
- 0 kudos
- 3684 Views
- 1 replies
- 0 kudos
pass a tuple as parameter to sql query
at_lst = ['131','132','133'] at_tup = (*at_lst,) print(at_tup) ('131','132','133')<div> <div><span>In my sql query, i am trying to pass this on a parameter, however, it doesn't work. <div> <div><div><div><span>%sql<br /><div><span>select * from ma...
- 3684 Views
- 1 replies
- 0 kudos
- 0 kudos
@Retired_mod I am writing sql using the magic command in the cell block, `%%sql`. Is there a way to pass a parameter in the query without using the `execute` method of the cursor object? Can you please share an example?
- 0 kudos
- 3129 Views
- 0 replies
- 0 kudos
Ingesting PowerBI Tables to databricks
Hi Community,I am looking for a way to access the Power BI tables from databricks and import them as a spark dataframe into my databricks notebook.As far as I have seen, there is a Power BI connector to load data from databricks into Power BI but not...
- 3129 Views
- 0 replies
- 0 kudos
- 2375 Views
- 0 replies
- 0 kudos
Changing StreamingWrite API in DBR 13.1 may lead to incompatibility with Spark 3.4
I'm using StarRocks Connector[2] to ingest data to StarRocks on DataBricks 13.1 (powered by Spark 3.4.0). The connector could run on community Spark 3.4, but fail on the DBR. The reason is (the full stack trace is attached)java.lang.IncompatibleClass...
- 2375 Views
- 0 replies
- 0 kudos
- 1146 Views
- 0 replies
- 0 kudos
Facing UNKNOWN_FIELD_EXCEPTION.NEW_FIELDS_IN_FILE
[UNKNOWN_FIELD_EXCEPTION.NEW_FIELDS_IN_FILE] Encountered unknown fields during parsing: [<field_name>], which can be fixed by an automatic retry: trueI am using Azure Databricks, and write with python code. Want to catch the error and raise. Tried wi...
- 1146 Views
- 0 replies
- 0 kudos
